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ABSTRACT 



A Study explored the application of reflective judgment 
theory to the types of problem solving faced by students in community service 
placements. It also tested the impact of different service-learning 
experiences on growth in reflective judgment over the course of a semester. 
Participants were 66 students in 6 colleges and universities. They were 
interviewed at the beginning and end of their service- learning semester using 
a problem-solving protocol based on issues with which they were dealing in 
their service. Although service alone did not appear to contribute to 
development of more complex reflective judgment, students who were in 
well- integrated and highly reflective service-learning classes were 
significantly more likely to demonstrate growth than those in moderate or low 
intensity service-learning experiences. (Appendixes contain 18 references, 2 
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Service-Learning and the Development of Reflective Judgment 



Abstract: In this study, we explore the application of reflective judgment theory to the 

types of problem solving faced by studaits in community service placements and test the 
impact of different service-learning ecperiaices on growth in reflective judgmait over the 
course of a semester. Participants were 66 students in 6 colleges and universities; they 
were interviewed at the beginning and aid of their SCTvice-leaming semester using a 
problem solving protocol based on issues they were dealing with in their service. While 
service alone did not appear to contribute to developmait of more complex reflective 
judgment, studaits who were in well integrated and highly reflective service-learning 
classes were significantly more likely to demonstrate growth than those in moda:ate or low 
intensity service-learning experiences. 



Cognitive Outcomes of Service-Learning 

"Service, combined with learning, adds value to each and transforms both" (Honna and 
Poulsen, 1989) This quote captures the core of widely held practitiona' belief about what is 
unique in service-learning programs which combine classroom learning with service work in the 
community. Learning improves the quality of service today and more importantly helps sustain it 
throughout a citizen's life; service transforms learning, changing inert knowledge to knowledge 
and compaencies that students can use in their communities. And this practitioner wisdom about 
effective learning is consistent with a long tradition of experiential learning theory from Dewey to 
modem cognitive scientists. 

This belief that service leads to better learning has led to a virtual explosion of service- 
learning programs (O'Brien, 1993) While the political support for sUch programs has grown, 
thae is very little empirical research to go along with the sodal and theoraical justifications for 
service-learning. The need to anchor programs in empirical evidence has led to a growing demand 
for research information about the effects of service-learning on cognitive outcomes. Most of the 
research has focused on the effects of savice on attitudes and values, and thae is good evidence to 
support the effectiveness of service-learning on students’ pasonal developmait and social 
responsibility. Thae has been less evidence of intdlectual impart. (Eyla and Giles, 1997) While 
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there is some limited evidence that service can increase conventional classroom knowledge 
acquisition (Markus, Howard and King, 1993), most studies have not shown an impact on 
increased factual knowledge. Indeed, one of the important research questions for the field is the 
idaitification of cognitive outcomes that are unique to or particularly enhanced by service-learning. 
(Giles and Eyler, forthcoming) 

It may be in the domain of complex problem solving rather than simple fact acquisition that 
service-learning makes its greatest contributions. The ability to think critically about complicated 
community problems is central to effective citizenship and also arguably more likely to develop 
when students address these issues in context. Service- learning is a perfect example of what 
Dewey describes as an educative experience i.e it involves activities which are interesting and 
awaken curiosity in students, which involve experience allows development over time and which 
are intrinsically important. (Dewey, 1933) When students as part of their coursework perform 
community service, they have the opportunity to address authaitic problems in the field and bring 
critical analysis to bear; service-learning encourages students to generate and answer real questions 
and helps them develop a nuanced understanding of issues in situational context. There is some 
evidence that service-leaming contributes to a more complex understanding of social problems. 

(Conrad and Hedin, 1980: Eyler and Halteman, 1981; Batchelder and Root, 1993; Giles and Eyler, 
1994) 

One measure of cognitive development with strong implications for 
understanding of community issues is what King and Kitchaier (1994)have termed 
‘reflectivejudgment’, a form of post formal reasoning. Embedded in the reflective 
judgment model of adult cognitive developmait is a series of increasingly complex 

cognitive skills that are integral to successful professional problem solving and a closely 
rdated series of increasingly complex assumptions about the nature of knowledge and 
justificatioa (King and Kitchener, 1994) More than two decades of longitudinal and 
cross-sectional research have indicated that with time and an ^jpropriate learning 
environment adolescents and adults develop skills for identifying, fiariiing, resolving and 
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re-addressing unstructured problems. Those with higher levels of reflective judgment are 
b^er able to assess the complexity of issues and to find, use and evaluate information 
more effectivdy than those with less well developed understanding of how to deal with 
conflicting sources of knowledge. Developmait of more complex reflective judgment is 
associated with the learning processes i.e. analysis of complec issues within realistic 
contexts, that are central to service-learning. It is particularly appropriate as a process for 
measuring the impact of service-learning because its focus is on ill-structured piroblems like 
those faced by citizais in their communities. 

This study is designed to test the propxisition that service-learning classes which are 
highly reflective and continuously challenge students to analyze social problems in the 
context of their expierience wth jjromote higher levels of critical thinkin g ability. Because 
reflective judgment theory is an attempt to understand increasingly adequate ways to deal 
with complex ill-structured problems, we have chosen to measure critical t hinkin g ability 
using this methodology. This study addresses the following questions: 

1. Win previous service, jirevious service-learning, age or gender affect the pro- 
test levd of reflective judgmait? 

2. Will students who participate in service-learning show significant growth in 
reflective judgmait compared to those students who do not pjarticipate ovct the 
course of a semester? 

3. Will students who participate in high intensity service-learning classes where 
reflection on the service is central to the day to day activities of the class show 
growth in reflective judgment compared to students whose service is less central to 
the daily activities of the class? 
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Reflective Judgment, Service-Learning and Ill-structured Problem Solving 

Reflective judgmait addresses the critical thinking skills needed by people coping 
with ill-structured problems. Well structured problems have clear goals and known 
constraints and their solution paths are well established; many dilemmas faced by technical 
experts have these characteristics. In contrast, the problems encountered by students in 
service-learning are similar to the problems oicountCTed by social scientists. They usually 
involve a problematic situation, such as poverty, homelessness or continuing spread of 
HTV in the population and their solution oftai lies in eliminating causes of the problem. 
Typically there is no consensus about the best solution to these complex social problems. 
They must be addressed under conditions of uncertainty regarding sources of adequate 
data, interpretation of that data, alternative solutions, constraints on possible solutions and 
the impact of various options. The ultimate goal is to make decisions in the face of 
complerity and uncertainty. Because information essaitial to solving the problem, such as 
causes or constraints is often unknown, the problem soIvct must often supply them from 
his or her own knowledge or experience or have complex strategies for acquiring new 
informatiotL Feedback about the efficacy of solutions is oftai not immediate and there is 
often no clear criteria for evaluating successful solutions to the problem. (Voss, Tyler and 
Yengo, 1983) 

Voss and his colleagues who have conducted research to explore differoices in 
social sciaice problem solving of experts and novices have found that experts tend to focus 
more heavily on the defining and framing of the problems, where novices tend to leap to a 
solutiotL Experts explore the knowledge of the domain to construct a problem 
representatiotL Experts also tend to generate solutions in which a large number of their 
statements are devoted to justifications for the solutions and consideration of alternatives. 
(Voss et al, 1983) This is paralld to the process of problem solving identified in the 
reflective judgmait literature. (Lynch, 1996) An exploratory sturty of student community 
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service volunteers found that the more experienced students showed a similar pattern in 
their approach to problem solving in the social sciaice research. They tended to be more 
elaborate and coherent in their representation of the community problem and to spend more 
time on problem analysis before exploring solutions. Their solutions were also more likely 
to be complex and involve several related subproblems. (Eyler, Root and Giles, 1997) 

Viable solutions to real problems that include a range of uncertainties result from the 
use of an increasingly complex set of cognitive skills evidence in a problem solving process 
that has several identifiable phases: idaitifying, flaming, resolving, and re-addressing 
unstructured social problems. (Lynch, 1996) Identifying the nature of the problem 
involves distinguishing between highly structured problems or subproblems, for which 
there is commonly agreed upon correct answers and unstructured problems that are fraught 
with uncertainties ranging flom the definition of the problem to the range and impact of 
various solutions. Failure to properly identify the nature of a problem i.e. mistaking an 
unstructured problem for a structured one, may cause one to inadequately explore the 
problem, which in turn can cause significant difficulties in articulating and justifying a 
solution. Inexperienced problem solva-s often focus on tangential aspects of the problem, 
especially those that might be considered more well structured, to the exclusion of the 
larger more difficult unstructured issues. They also taid to slight this part of the process 
and move directly to the solution phase as noted in the expert novice literature skipping the 
flaming process altogether. 

Framing an unstructured problem means exploring the problem and related 
information as thoroughly as resources permit. Inadequate framing skill s can lead students 
to oversimplified constructions of complex social problems resulting in lack of attention to 
a) multiple perspectives, b) altonative interpr^tions of the full range of relevant 
information and minimirin g the efFprts of unfounded biases. 

Resolving an unstructured problem means idaitifying, articulating and strongly 
justifying a solution as most viable. After flaming the problem, one can t hink more clearly 
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about possible solutions. Many people find it somewhat difficult to complete tasks that 
require strong justifications for their own opinions and effective counter arguments for 
other options. Only through adequately exploring the problem and judging among 
competing options as objectively as possible can we hcpe to put forth the best possible 
solution with the available information. 

Re-addressing or -re-solving an unstructured problem. Once one has 
recognized the unstmctured nature of a community problem and expended effort in 
framing, articulating and justifying a particular resolution as most viable, it is important to 
recognize the limitations of that resolution. In many cases, citizens must devise strategies 
for gathering new information and evaluating the results of previous solutions. (Golub, 
1997; Schon, 1983) When important new information is obtained, a reconsideration of the 
problem is needed. As the process is repeated over time, solutions are likely to become 
more adequate. Figures 1,2, and 3 come from Lynch and Wolcott’s application of 
reflective judgment theory to the task of professional problem solving; the same process 
and the same types of challaiges to students articulated in Figure 3 are equally applicable to 
students tackling ill-structured social problems in their service-learning classes. (Lynch and 
Wolcott, 1997) 

The reflective judgmait model describes seven qualitatively different sets of 
assumptions about knowledge. Tables 1 and 2 drawn from Kitchener and King (1985) 
summarize the key assumptions in levels 3 through 7. The types of reasoning described in 
levels 1 and 2 are rarely found among college students. The increasing cognitive 
complexity associated with each stage builds on the previous one and builds a foundation 
for the next with appropriate challenge and support. Students move from a tendaicy to 
view matters in simple black and white toms, through assuming that truth is relative and a 
matter of personal experience or opinion to a complex process for wei g hin g 
competing perspectives and integrating information from varied sources. 
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Community experience, where students face the complexities of social problems 
and are someiimes confronted with perspectives as well as experiences that are new to 
them, provide material for challenging reflection in the classroom. In a highly reflective 
service-learning class, students should be challaiged to understand that the problem is ill- 
structured, to clearly frame it, and to seek out and evaluate a variety of resources and 
perspectives as th^ seek to resolve it. Such experiaices will encourage students to 
sharpen their problem solving skill s and refine their assumptions to become more adequate 
for dealing with complex problems. Service experiaice without this continuous 
intellectual challenge may have a less visible impact on how students think about problems. 
This study is an exploratory effort to test the plausibility of this notion. 

Methods of the Study. 

Sample. Sixty six students who were enrolled in courses with community service 
components were interviewed at the beginning and end of the spring semester of 
1996. These courses were located at six colleges and universities and included both 
private and large public institutions; included were: University of Colorado, Indiana 
University- Purdue University at Indianapolis, Michigan State University, CaUfomia State 
College at Monterey Bay, Nazar^ College and Vanderbilt University. In two courses 
where service was an option rather than a requirement of the course, a sample of students 
who did not select the service option were interviewed as a control group. There were 16 
students in this control group. 

InstPimpTit. The interview protocol included general questions about community 
service and a series of questions patterned on the classic Reflective Judgment Intaview 
(King and Kitchener, 1994 ) but designed to address community problems. The problem 
focus for each reflective judgment interview was a social problem related to the students’ 
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particular service that semester. The interviews lasted approximately 50 minutes and the 
same interviewer conducted the pre and post interview for each studait. 

Coding Interview Data Interviews were transcribed and scored by a coder trained 
in the reflective judgmait coding m^hod. While it was not possible to disguise whether 
interviews were pre or post because of internal cues, the coder was not aware if particular 
transcripts represaited students from the control group or the highly reflective or less 
intense service-learning experiences. Transcripts were ‘shuffled’ and interviews from 
particular times and schools were not coded sequentially. An expert coder who trains RJ 
coders scored a subsample to check for reliability. The coders had an agreement of .8 and 
discrepancies were resolved through conferencing. 

Data Analysis. Data were analyzed using hierarchical multiple regression with 
controls for age, gender, previous community service, and previous service-learning. The 
first analysis examined the impact of these background charactaistics on the pre-test 
measure of reflective judgment. The second analysis examined the impact of participation 
in service. The third analysis used only the 50 students who participated in service and 
©camined the effects of an intensive reflective service-learning experience compared to 
participation in a less intensive experience. Intensive classes were those where the service 
was required and caitral to the daily class discourse; these classes focused on analyzing 
the service e^)erience and relating it to the subject matter. Less intensive classes made 
service an option or were less likely to focus class activities on analysis of the experience. 

Findings 

Predicting Pre Test Reflective Judgment Scores. The average pretest RJ score for 
all studmts was 4.08 which is slightly higher but consistent with scores for college 
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students reported in the literature. (King & Kitchener, 1994) As might be expected with a 
developmental construct, the best predictor of reflective judgment before the service 
semester was the students’ ages. In fact age was the only predictor of pretest RJ with a 
b«a of .45. Gender, previous service-learning and previous sCTvice were all not 
significant pre test predictors of RJ. 

Impact of Service on Reflective Judgment Scores. In the combined sample of 
control and service-learning studaits, service was not a predictor of growth in RJ over the 
course of the semester. Most students, whether service-learning students or controls, were 
stable in their RJ scores over this period of time. 

Impact of Highly Reflective Service-Learning on Growth in Reflective Judgment. 
While most students did not show growth in their RJ scores over the relatively brief course 
of a semester, about 20 % of those participating in service-learning did. For the control 
group the mean pre test and post test scores were 3.96 and 3.94 respectively. When we 
compared the impact of moderately and highly integrated service-learning experiences the 
results were somewhat differait. While the students in the moderately intense service 
experience averaged 4.07 before and 4.08 after their service semester, students in the 
intensive service-learning classes showed more growth. The RJ average before service was 
4. 16 and after service was 4.41. In the regression analysis, high integration of the service 
and learning was a significant predictor of growth with a beta Of .23. Interestingly, 
previous service-learning classes were also a significant predictor of growth with a beta of 
.20 suggesting that the effects of service-learning may be cumulative; this earlier experience 
was not a predictor of RJ 1 at the pre test. 

Conclusions 

This study provides some modest support for the view that highly reflective 
service-learning expaiaices have an impact on students’ ability to deal with complex 
unstructmred problems. It suggests that if developmait of problem solving sldll is a goal of 
the curriculum that participation in community service may not be enough; students also 



Eyler, Giles, Lynch & Gray Reflective Judgment 



need to be challoiged to think about the implications of what they are experiencing and to 
apply subject matter learning to the field and vice versa. Service needs to be tightly linked 
to learning and reflection is that link . 

If instructors are to design courses to enhance the ability of students to deal with 
unstructured problems, thoi it will be helpful for them to understand the types of questions 
and challenges that students at various developmaital stages will find useful. The 
questions in Figure 3 suggest some guides to reflection on understanding these complex 
issues. And further research to explore the impact of building this focused reflection into 
the course will be helpful. This study was exploratory and used courses of varying 
degrees of reflective intensity as they occurred in the field. Further research would 
profitably focus on experimenting with contrasting approaches to reflection including those 
more tightly tied to problem solving skills embedded in the reflective judgment model such 
as the sequence suggested in Figures 2 and 3. 

It is also helpful to faculty to be able to monitor student growth and to arrive at a 
rough assessment of where students are. The interview process is useful for conducting 
research, but much too cumbersome to use as an assessment tool in the classroom. 

Wolcott and Lynch (1997) experimented with using a writtai reflective judgmait essay in 
their work with an accounting class. Students were presaited with a challenging 
accountancy problem and asked to discuss both what they would do and how they arrived 
at their conclusions. There responses were assessed according to the assumptions and 
arguments identified in Tables 1 and 2. This essay could then serve as a baseline and be 
repeated latCT in the semester; since it is focused on the course subject matter it is simply 
part of the assignmoit structure of the course. We are currently piloting a "amilar essay 
assignment in a service-learning class related to public policy. Such an assignment helps 
the instructor identify issues and questions to pursue during class discussion and also helps 
aeate an expectation on the part of students that they will concern themselves with both the 
substance and process of their decisions about unstructured social problems. 
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Assumptions About Knowledge Related to 
Levels 3 Through 7 of the Reflective Judgment Model 
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Source: Kitchener (1986, 78-79). 



TABLE 2 



Approaches to Unstructured Problems Based on 
Levels 3 Through 7 of the Reflective Judgment Model 



Level 


Recognizing the 
Nature of a Problem 


Framing an 
Unstructured Problem 


Resolving an 
Unstructured Problem 


3 


• All problems perceived as 
well structured 

• Looking for the right answer 


• Attempts rather mechanical 

• Unable to frame in light of 
uncertainties 

• May respond with expressions 
of confusion or futility 


• Unable to reason logically from 
evidence to conclusions 

• Tendency to fall back onto 
whim or prior beliefs 

• Opinions classified as correct 
or incorrect, or equally valid if 
imcertainty is acknowledged 


4 


• Distinguishes between well 
structured and unstructured 
problems 

• Likely to attribute 
imcertaindes to a limited 
range of reasons 


• Respond to problem rather 
holistically; limited ability to 
break it down or place it in its 
larger context 

• Can identify evidence and 
various perspectives, but ability 
to logically organize this 
information is limited 


• May appear reluctant to 
examine own views objectively 

• Attempts at examining 
evidence and competing views 
may reflect quantitative rather 
than qualitative approaches 

• Often argues that solutions are 
equally valid 


5 


• Understands the relative 
nature of unstructured 
problems when viewed from 
different perspectives 


• Significant efforts to articulate 
a detached, balanced view of 
the problem from different 
perspectives or contexts 


• Uses evidence to reason 
logically within a given 
perspective 

• Justifies beliefs by rules of 
inquiry for a particular context 

• May seem reluctant about 
articulating and defending a 
solution as most viable 


6 


• Understands that 
unstructured problems 
require coming to personal 
conclusions, based on 
evaluations across 
perspectives. 


• Frames problems so that more 
generalized principles, 
concepts or goals subsume 
different perspectives 


• Able to make well-founded 
judgments about which 
solutions are more viable than 
others 

• Likely to base opinions on 
evaluated beliefs of experts or 
the pragmatics of situation 


7 


• Understands that new and 
better knowledge can be 
constructed through 
synthesis of existing views 
and evidence 

• On-going, cumulative nature 
of knowledge m^ be 
articulated 


• Addresses large, unstructured 
problems with complex, 
flexible frameworks that can 
provide new insight 

• Breaks issues into parts that 
can be addressed in appropriate 
ways while maintaining 
rational view of greater 
problem context 


• Systematically uses processes 
of critical inquiry to work 
through complex, unstructured 
problems 

• Argues convincingly using 
complex, coherent discussion 
of own perspective, including 
strengths and limitations 



Source: Kitchener and King (1985). 
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Source: Lynch and Wolcott, 1997 “Optimizing Professional Problem Solving 



RE-ADDRESSING OR 
RESOLVING 

AN UNSTRUCTURED PROBLEM 
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